Method and apparatus for planning computer-aided diagnosis

ABSTRACT

The invention provides a method and apparatus for classifying a region of interest in imaging data, the method comprising:
         calculating a feature vector for at least one region of interest in the imaging data, said feature vector including features of a first modality;   projecting the feature vector for the at least one region of interest in the imaging data using a decision function to generate a classification, wherein the decision function is based on classified feature vectors including features of a first modality and features of a second modality;   estimating the confidence of the classification if the feature vector is enhanced with features of the second modality.

FIELD OF THE DISCLOSURE

The disclosure relates to computer-aided diagnosis (CAD). The disclosurealso relates to a method and a platform or system for using machinelearning algorithms for CAD.

BACKGROUND OF THE DISCLOSURE

Advances in computed tomography (CT) allow early detection of cancer, inparticular lung cancer which is one of the most common cancers. As aresult, there is increased focus on using regular low-dose CT screeningsto ensure early detection of the disease with improved chances ofsuccess of the following treatment. This increased focus leads to anincreased workload for professionals such as radiologists who have toanalyze the CT screenings.

To cope with the increased workload, computer-aided detection (CADe) andcomputer-aided diagnosis (CADx) systems are being developed. Hereafterboth types of systems will be referred to as CAD systems. CAD systemscan detect lesions (e.g. nodules) and subsequently classify them asmalignant or benign. A classification need not be binary, it can alsoinclude a stage of the cancer. Usually, a classification is accompaniedwith a confidence value as calculated by the CAD system.

CAD systems typically follow a number of general steps. First, the inputimaging data is segmented, for example to distinguish lung tissue fromthe background signal. Then, regions of interest are identified, forexample all lung tissue with nodule-like forms in them. For each regionof interest a number of input values is calculated, the so-calledfeature vector. This feature vector is used as input in a decisionfunction, which projects the feature vector to a classification.

Hereafter the term “model” will be used to indicate a computationalframework for performing one or more of a segmentation and aclassification of imaging data. The segmentation, identification ofregions of interest, and/or the classification may involve the use of amachine learning (ML) algorithm. The model comprises at least onedecision function, which may be based on a machine learning algorithm,which projects the input to an output. For example, a decision functionmay project a feature vector to a classification outcome.

A problem with diagnosis in general is that there are many possibletests to subject the patient to. However, not every test is equallyuseful and there are medical, practical, and financial limits to thenumber of tests that can be used.

SUMMARY OF THE DISCLOSURE

It is an object of this disclosure to provide a method and apparatus forclassifying a region of interest in imaging data which addresses theabove drawbacks.

The disclosure provides a method for classifying a region of interest inimaging data, the method comprising:

-   -   calculating a feature vector for at least one region of interest        in the imaging data, said feature vector including features of a        first modality;    -   projecting the feature vector for the at least one region of        interest in the imaging data using a decision function to        generate a classification, wherein the decision function is        based on classified feature vectors including features of a        first modality and features of at least a second modality;    -   estimating the confidence of the classification if the feature        vector is enhanced with features of the second modality.

Further examples are disclosed in the attached claims and described inreference to the figures below.

The disclosure thus allows planning of tests which are to provide inputfor CAD. By aggregating (training) data, the CAD models can be generallyimproved. With sufficient training data for various modalities, itbecomes possible to reliably determine, given that only a part of theavailable features (say, only the features corresponding to modality 1but not the features of modality 2 and modality 3) what the likelihoodwill be that a measurement of modality 2 or modality 3 will addinformation that will make the classification significantly morereliable. In case it is determined that, say, modality 2 will add nosignificant reliability, the test for modality 2 can be omitted, savingtime and costs. Accordingly, the disclosure allows professionals tooptimally use available means (technology, money, patient availability)in order to obtain the most reliable diagnosis.

The disclosure further provides a computation platform a computationdevice, configured to implement the above described methods. Thedisclosure further provides a system of a computation platform and auser terminal, configured to implement the above described methods. Thedisclosure further provides a non-transitory computer readable mediumcomprising computer instructions for implementing the methods accordingthe disclosure.

BRIEF DESCRIPTION OF THE FIGURES

Embodiments of the present disclosure will be described hereinafter, byway of example only, with reference to the accompanying drawings whichare schematic in nature and therefore not necessarily drawn to scale.Furthermore, like reference signs in the drawings relate to likeelements.

FIG. 1 schematically shows an overview of a workflow according toembodiments of the disclosed subject matter.

FIG. 2 schematically shows a flow chart according to embodiments of thedisclosed subject matter.

FIG. 3 schematically shows a flow chart for a model calculationaccording to embodiments of the disclosed subject matter.

FIG. 4 schematically shows a flow chart for specifying details of amodel calculation according to embodiments of the disclosed subjectmatter.

FIGS. 5A and 5B schematically illustrate the functioning of a SupportVector Machine model according to embodiments of the disclosed subjectmatter.

FIG. 5C schematically illustrates the functioning of a machine learningmodel according to embodiments of the disclosed subject matter.

FIG. 6A schematically shows information flows between a workstation anda platform according to embodiments of the disclosed subject matter.

FIGS. 6B and 6C schematically illustrate a method for adding additionaldata according to embodiments of the disclosed subject matter.

FIGS. 7A and 7B schematically illustrate a method for calculating anensemble result according to embodiments of the disclosed subjectmatter.

FIGS. 8A-8D show example screens for setting model parameters accordingto embodiments of the disclosed subject matter.

FIG. 9 shows an example screen of a virtual radiologist according toembodiments of the disclosed subject matter.

FIG. 10 shows an example screen for image features selection accordingto embodiments of the disclosed subject matter.

FIGS. 11 and 12 shows a flow diagram for a model calculation accordingto embodiments of the disclosed subject matter.

FIGS. 13 and 14 schematically show a flow diagram for estimating modelconfidence according to embodiments of the disclosed subject matter.

FIGS. 15A-15F schematically show various scenarios involving differentmodalities according to embodiments of the disclosed subject matter.

FIG. 16 shows a flow diagram for making patient data available accordingto embodiments of the disclosed subject matter.

DETAILED DESCRIPTION

In the detailed description, captions are merely used to organize theinformation in a fashion to facilitate reading. These captions are notto be understood as indicating separations between distinct embodimentsor distinct aspects of the disclosure.

Computation Platform

FIG. 1 schematically shows an overview of a workflow according toembodiments of the disclosed subject matter. A patient is scanned inscanning device 10. The scanning device 10 can be any type of device forgenerating diagnostic image data, for example an X-Ray device, aMagnetic Resonance Imaging (MRI) scanner, or any general ComputedTomography (CT) device. Of particular interest are low-dose X-Raydevices for regular and routine scans. The various types of scans can befurther characterized by the use of a contrast agent, if any.

In the following, the example of a CT device, in particular a CT devicefor low dose screenings, will be used. However, this is only exemplary.Aspects of the disclosure can be applied to any instantiation of imagingmodality, provided that it is capable of providing imaging data. Adistinct type of scan (X-Ray CT, low-dose X-Ray CT, CT with contrastagent X) can be defined as a modality.

The images generated by the CT device 10 (hereafter: imaging data) aresent to a storage 11 (step S1). The storage 11 can be a local storage,for example close to or part of the CT device 10. It can also be part ofthe IT infrastructure of the institute that hosts the CT device 10. Thestorage 11 is convenient but not essential. The data could also be sentdirectly from the CT device 10 to computation platform 12.

All or parts of the imaging data is then sent to the computationplatform 12 in step S2. In general it is most useful to send allacquired data, so that the computer models of platform 12 can use allavailable information. However, partial data may be sent to savebandwidth, to remove redundant data, or because of limitations on whatis allowed to be sent (e.g. because of patient privacy considerations).The data sent to the computation platform 12 may be provided withmetadata from scanner 10, storage 11, or further database 11 a. Metadatacan include additional data related to the imaging data. For examplestatistical data of the patient (gender, age, medical history) or dataconcerning the equipment used (type and brand of equipment, scanningsettings, etc).

Computation platform 12 comprises one or more storage devices 13 and oneor more computation devices 14, along with the necessary networkinfrastructure to interconnect the devices 13, 14 and to connect themwith the outside world, preferably via the Internet. It should be notedthat the term “computation platform” is used to indicate a convenientimplementation means (e.g. via available cloud computing resources).However, embodiments of the disclosure may use a “private platform”,i.e. storage and computing devices on a restricted network, for examplethe local network of an institution or hospital. The term “computationplatform” as used in this application does not preclude embodiments ofsuch private implementations, nor does it exclude embodiments ofcentralized or distributed (cloud) computing platforms.

The imaging data is stored in the storage 13. The central computingdevices 14 can process the imaging data to generate feature data asinput for the models. The computing devices 14 can segment imaging data.The computing devices 14 can also use the models to classify the(segmented) imaging data. More functionality of the computing devices 14will be described in reference to the other figures.

A work station 15 for use by a professional, for example a radiologist,is connected to the computation platform 12. Hereafter, the terms“professional” and “user” will be used interchangeably. The work station15 is configured to receive data and model calculations from thecomputation platform, and to send instructions and feedback to thecomputation platform 12. The work station 15 can visualize received rawdata and model results.

In step S3, the professional selects a basal model (or in general:specifies model parameters) for use in a calculation. More exemplarydetails concerning basal models are provided in reference to FIG. 9.Based on the entered model parameters, in step S4 the platform 12generates the model (if needed—the model may be already cached),performs the needed calculations for training the model (ifneeded—training data for the model may already be available in thecomputation platform 12), and applies the model to the imaging data thatwas received in step S2. In general, the computation platform will usestored results for calculations that have been performed earlier (i.e.calculated image features, model training data) and only perform thecalculations it has not done before. This way, the professionalaccessing the computation platform 12 using the work station 15 can havea fast response to his or her instructions.

The result of the model calculations, for example a segmentation of theimaging data and corresponding classification, is sent to theprofessional in step S5. The received data is visualized on the workstation 15. The professional will examine the results and preparefeedback in step S6. Feedback may for example be that, in theprofessional's opinion, the presented classification is correct orincorrect. Other types of feedback are also available in exemplaryembodiments, for example: the professional can confirm or correct thebasic classification (e.g. malignant or benign) and also add furtherinformation, for example a stage of the cancer in case of a malignclassification. In this manner, the feedback information can be used toenrich the classified feature vectors so that at a later stage moresophisticated models can be trained.

The feedback from step S6 is sent to the computation platform 12. Instep S7, the computation platform 12 incorporates the feedback in itsown data. For example, if the feedback is of the correct/incorrect ormalignant/benign type, the model results and the feedback can be addedas ground truths for further training. Along with the feedback, thesource of the feedback may also be stored. That makes it possible totrain future models using only feedback from selected sources. Forexample, the professional can request models that are only trained usinghis own data or data from close colleagues (e.g. “trusted data”).Instead or in addition to this, the feedback can be used incrementallyadjust the decisions functions of the model. The feedback can be usedonly in one or more selected decision functions, again to insure thatmodels are trained using data from known and trusted sources.

FIG. 2 provides an overview of a workflow of using a model and thecomputation platform 12 from the work station 15. In step 20 thecomputation platform 12 receives imaging data from local storage 11 ordirectly from scanning device 10. The imaging data is processed in step21 (more possible details of the processing step are described inreference to FIG. 3). The professional at the work station 15 sendsmodel parameters to the computation platform 12 in step 22. These modelparameters can form a “basal model” for use on the imaging data. Themodel parameters can comprise parameters for the model that will beused. For example the model parameters can specify which training datais to be used or which basic model is to be used.

With the term “basic model” is understood what the core model algorithmis, for example one of support vector machine, decision tree, decisionforest, adaboost, boosted stumps, etc.

In step 23 the model, including a decision function, is generated (orretrieved, in case the model parameters correspond or refer to apreviously generated and stored model) by the computation platform 12.After the model is generated or retrieved, in step 24, the model is runon the imaging data. The results of the model calculation are then sentto the workstation for evaluation. The results of the model calculationcan comprise e.g. a segmentation, a determination of one or more regionsof interest, a classification of one or more regions of interest, or acombination of these. In step 25 the professional at the workstation 15evaluates the model results which are presented visually on theworkstation. The professional may decide that adjustments to the modelor to the model parameters are needed in step 26. If adjustments areneeded, new input parameters are input by the professional and sent bythe workstation to the computation platform 12. The computation platform12 will then re-apply step 23 making only new calculations wherenecessary by the new parameters.

This process will continue until the professional is satisfied with theresults from the model. The skilled person can then send his feedback onthat model and on the model calculation results to the client platform12. The computation platform 12 will then record the final modelparameters used in the model calculation in step 27.

The computation platform 12 may also store classification or diagnosticfeedback from the professional. For example the computation platform maystore a positive or a negative diagnosis of a certain condition (step28). The (processed) imaging data used in the calculation can thenbecome training data with associated ground truth for future modeltraining.

In a variant, the feedback from the professional is used toincrementally update one or more selected decision functions of themodel.

Models

The examples disclosed in reference to FIGS. 3, 4, 5A and 5B focus onthe use Support Vector Machines (SVMs) as the underlying basic model ofthe calculation. However, the disclosure is not limited to SVMs. Furtheron, brief explanations about the application of this disclosure todecision trees and boosted stumps will be provided. The skilled personwill be able to apply the principles behind the examples in thisdisclosure to other types of models.

FIG. 3 schematically shows a flow chart for a model calculationaccording to embodiments of the disclosed subject matter. The steps31-36 can be seen as an exemplary implementation of the steps 20, 21,23, 25 in FIG. 2, using a Support Vector Machine (SVM) as the basicmodel. However, for other types of basic models, the steps 31-35 mayapply as well, with only the final classification 36 being dependent onthe type of basic model.

In step 31, imaging data is acquired, for example by reading it from astorage or by receiving it from a scanning device. The imaging data issegmented in step 32. In the segmenting step, areas may be identifiedthat correspond to certain organs (e.g. lungs, heart, breast, etc).After the segmentation, regions of interest may be determined in step33. This can be done manually (e.g. an operator indicates which regionsare to be investigated further) or automatically, for example usingtemplate matching to identify candidate nodules in an image followed byvariational calculus to determine the outlines of the candidate nodules.Data corresponding to the regions of interest may be stored as processedimaging data.

It is possible that some of the steps 31-33 are already performed beforethe imaging data is delivered to the computation platform 12, forexample as post-processing in a scanning device. In that case, thesesteps can be omitted.

Once the regions of interest or, in specific cases, the candidatenodules are selected or detected, the item of interest (the nodule) issegmented in step 34. The item of interest (nodule) is separated fromits surroundings.

Features of said regions or nodules are calculated in step 35. Intypical modules, a range of features is calculated. Many features areavailable to the skilled person, for example symmetry features,circularity, spikulation, etc. Also high level features can be included,such as age, gender, and body mass. All feature values together form afeature vector. The calculated features may be stored as processedimaging data.

By storing the regions of interest (candidate nodule) imaging dataand/or the calculated feature vector data as processed imaging data, itmay not be necessary to store the entire imaging data that was acquiredin step 31. This can be advantageous to save storage space or if storingsuch patient data is not allowed.

The product of any one or combination of the steps 31, 32, 33, 34, 35can be stored for future reference. The advantage of storing data laterin the chain 31-35 is that generally less storage space is required andthere may be fewer data privacy concerns. A disadvantage is a loss offlexibility. For example, if the result of the candidate noduleselection (step 33) is stored, after which the original (step 31) andsegmented (step 32) imaging data is discarded, then it is not possibleto later to use an improved candidate nodule detection algorithm. Fordifferent applications, different choices may be made concerning whichdata is stored.

Finally, in step 36 each region of interest or candidate nodule isclassified by the Support Vector Machine. Typical outcomes of such aclassification are “malignant” or “not malignant” (a binary outcome).However, it is also possible to design SVMs to have more than 2 possibleoutcomes (for example, including a stage of the cancer in case of amalignant classification) or even a floating point value outcome. Theclassification step 36 will be explained in more detail in reference toFIGS. 5A and 5B.

FIG. 4 schematically shows a flow chart for specifying details of amodel calculation according to embodiments of the disclosed subjectmatter. The steps 41-45 can be seen as selections that the workstation15 user makes in step 22 of FIG. 2 and supplies to the computationplatform 12 as model parameters.

An important aspect of Machine Learning algorithms such as SupportVector Machines is the selection of data points that is used fortraining the ML algorithm. In general, more data points improve thequality and reliability of the ML algorithm's outcome. At the same time,noisy or erroneously qualified data points may detract significantlyfrom the quality and reliability. Therefore, it is important to only usequality data points from a known source.

In step 41, the data points are selected. Typically, the operator of thework station selects the data points by identifying which sources are tobe used for data points. For example, the operator may indicate thathis/her previous diagnoses is to be used to provide data points. Theoperator may indicate that only diagnoses for patients with certainconditions (e.g. age, gender) is to be used. Alternatively oradditionally, the operator may indicate that data points from a certain(prestigious) doctor or medical centre are to be used. It is envisagedthat individuals or institutions who specialize in certain diagnoses,will make their diagnoses available as usable data points in thecomputation platform 12. This allows operators in whose practice acertain diagnosis is not frequently made, to obtain sufficient datapoints from a known and trusted source.

In step 42, a selection is made of the features to be used in theclassification. Not every feature calculated in step 35 needs to be usedin the classification step 36. For most Machine Learning applications,there are a number of core features that are required. Selecting thefeatures is an important part of defining or tuning a model. Again, itis envisaged that an operator may select a feature subset by referringto a subset as used or defined by a leading individual or medicalinstitution.

The kernel selection in step 43 is a selection that is appropriate forSVMs. The operator may pick a specific kernel, select a default kernel,or refer to kernel as selected by a certain individual or institution.This also holds for the soft-margin/penalty parameter selection in step44. This parameter determines how much “slack” a Support Vector Machinecan use to allow for noisy data points or partially overlapping sets.

The user may determine a cross-validation strategy in step 45. In across-validation strategy, a subset of the data points can be reservedfor checking the consistency of the model. This helps to improve thereliability. Again, the operator can make his or her own selection,request a default value, or refer to a choice made by somebody else.

Once all parameters are set, the model can be generated in step 46. Ifthe basic model is a SVM, the generation step will calculate thehyperplane(s) separating data points from different categories (moredetails in FIGS. 5A and 5B). In general, the generation step 46 willinvolve using all available data points to train the algorithm topredict outcomes for new, unclassified, data points. With the generatedmodel, the cross-validation strategy of step 45 can be performed.

The outcome of the cross-validation can be a reliability indication. Thereliability indication can provide information on the expectedpercentage of false positives, sensitivity, etc, depending on thepurpose of the model and the meaning of the labels used in theclassification. If the reliability is considered too low, the model canbe improved, for example by adding extra data points. An operator mayfor example first attempt to train a model using only his own datapoints in step 41. If it then turns out in step 47 that the reliabilityis insufficient, the operator may go back to step 41 and add data pointsfrom a trusted source to improve the statistical robustness of themodel. This will be shown in more detail in reference to FIGS. 5A and 5Bbelow.

FIGS. 5A and 5B schematically illustrate the functioning of a SupportVector Machine model. For the sake of simplicity, an example SVM usingonly two features (feature 1 and feature 2) is used. In practice, thefeature vector will usually have far more than two components.

In both figures, the squares 121, 127 indicate data points (in thetwo-dimensional feature space formed by feature 1 and feature 2coordinates) with a certain label, e.g. +1. The circles 122, 128indicate data points with another label, e.g. −1. Again for the sake ofsimplicity only two types of labels are used in the present example. Onemay for example indicate “malignant” and the other “benign”. The skilledperson will be aware that SVMs can be generalized for any number oflabels or even for continuous values.

The triangle 123 indicates a data point that is to be classified (either+1 or −1 in the present example). For the present 2D example, an SVMtakes the labelled data points 121, 127, 122, 128 into account andattempts to define a line 124 separating the points with differentlabels so that the gap, defined as the distance between lines 125 and126 which run parallel to line 124 and which run through the squares andcircles respectively nearest to line 124, is maximal. The line 124determines the decision function of SVM. By determining the position ofthe triangle 123 relative to line 124, the label of triangle 123 can bepredicted. The larger the distance of the triangle from line 124, themore confident the prediction of the label will be.

The situation in FIG. 5A is an exemplary result of an SVM calculationwith relatively few data points (in this case 4, 2 for each label). TheSVM can find a line 124 and corresponding gap which successfullyclassifies all 4 data points. However, it is easy to see that manydifferent lines can be drawn to separate the 2 data points 121 from the2 data points 122, and the line 124 may thus not be the correct one. Inthis situation, the triangle 123 is very close to the found line 124,and the classification of triangle 123 (in this example as a +1 datapoint, belonging in square group 121) is quite uncertain.

In FIG. 5B, the original set of data points 121, 122, 123 of FIG. 5A ismaintained, but additional data points 127 (+1 group) and 128 (−1 group)have been added. This can correspond to the case wherein the operator instep 41 of FIG. 4, selects additional data points from a trusted sourceto improve the accuracy of the SVM. Now the line 124 calculated by theSVM to separate sets 121, 127 from 122, 128 is much more reliable. Thetriangle 123 is in this case classified as +1 (group 121, 127) with muchmore confidence.

If the data point 123 is classified by a professional and added to thedata set, it is possible to update line 124, and thereby the decisionfunction of the SVM, incrementally to account for the added data point.For SVMs, and indeed for many other ML algorithms, a full retrainingusing the data points 121, 127, 122, 128 is not needed. The addition ofpoint 123, if it is close to the so-called decision boundary, will“nudge” lines 124, 125, 126 which make up the decision function and thegap by a small amount, which can be readily calculated and whichstrongly depends on the distance of point 123 to the line 124. If thepoint is far from the decision boundary, it will typically have anegligible effect on the decision function. Incrementally updating thedecision function to accommodate a new data point has the advantage thatthe time-consuming retraining of the entire data set is not needed.

FIG. 5C schematically summarizes the functioning of a SVM or a machinelearning model in general. One or more sets of classified data points 51are indicated as training data (e.g. one set is the set 121, 122 of FIG.5A and another set is the set 127, 128 of FIG. 5B). The classified datapoint comprises at least the feature vector and the classification. Theinput 52 is the feature vector of the region of interest of the imagingdata. This can be the point 123 of FIGS. 5A and 5B. The model 54 isfurther determined by the model parameters 53.

The model 54 has a decision function which projects feature vector 52 toa classification. The outcome 55 of the model 54 is the classificationand, optionally, a confidence value for the classification. Theconfidence value can indicate how reliable the outcome is. Feedbackinformation 56 based on the outcome can be added, together with thefeature vector 52 which was used as input, as a new classified datapoint in data point set 51. This way, the model can be updated byincrementally updating the training set. Alternatively (oradditionally), the feedback information 56 can be directly added to themodel 54, to incrementally update the decision function to take thefeature vector 52 feedback 56 into account. That way, a full retrainingof the model using the updated training set 51 is not needed.

The above explanation in reference to SVMs can also be applied to othertypes of models. For example, decision trees can also be incrementallyupdated. First, it is checked if the decision tree already generates thecorrect classification for the newly added sample. If so, no furtheradjustment is needed. If it does not generate the correctclassification, it is generally possible to determine the branch were awrong decision is made and to update said branch. The same approach willwork, mutatis mutandis, for boosted stump models. In general, it will bepossible to incrementally update a model to introduce a new sample pointin the training data, without actually retraining the model using theentire training data set.

Processing

Returning now to the architecture as described in reference to FIGS. 1and 2, FIG. 6A schematically shows information flows between aworkstation 15 and a platform 12 according to embodiments of thedisclosed subject matter.

The storage device 13 of computation platform 12 stores in number ofdifferent data types, of which some examples are given in FIG. 6. Thestorage device 13 can store pre-recorded (and usually pre-processed)imaging data 60, which has previously been received as imaging data 70from an imaging data source, such as scanning device 10 or local storage11. As was mentioned before, the product of any one or combination ofthe steps 31, 32, 33, 34, 35 of FIG. 3 can be stored for futurereference as processed data 60. In addition, the storage 13 storesmetadata 61 (associated with the processed data 60), which has beenpreviously received from metadata source 11 a or which has been manuallyentered. Also associated with the processed data 60 and the metadata 61,is feedback information or ground truth data 62. The feedbackinformation can include information about the type of diagnosticinformation that is present in associated processed data 60. Themetadata 61 associated with processed data 60 includes information aboutthe patient and the patient's condition and about the type of scannerdevice 10 and the scanner modality used in obtaining the imaging data.The storage device 13 also include pre-defined models (e.g. sets ofmodel parameters) 63. Furthermore, the storage 13 can store cache data64 used by the modules of the computation means 14 of the platform.

The computation device 14 can include several modules, in part dependingon the type of basic model that is used: a data selector 65, a featureextractor 66, an image segmenter 67, a training module 68, and acalculation module 69. These modules may be logical modules, implementedas one or more sets of computer executable instructions to be executedon a processing unit. The data selector 65 can retrieve processed datafrom the storage 13. The feature extractor 66 is configured to receiveprocessed data from the data selector and/or the image segmenter and tocalculate selected features in the received data. The calculatedfeatures may be stored in the calculation cache 64 for future reference.The image segmenter 67 can segment the image into distinct areas. Theimage segmenter 67 can make use of domain knowledge, so that for examplea dedicated image segmenter module is provided for lung CT imaging data,and a different image segmenter for e.g. breast image data. The imagesegmenter 67 may also isolate objects in regions of interest, forexample suspect nodules.

The training/updating module 68 can receive imaging data and featurevectors, and perform a machine learning training procedure to create afully usable model. In addition, the training/updating module can takean existing (trained) model and incrementally update the model to add anew classified feature vector, e.g. based on feedback received inconnection with a previous calculation on a feature vector.

Intermediate results of the training/updating may again be stored in thecalculation cache 64 of storage device 13. After training or updating,the model can be stored in the pre-defined model storage 63. Thetraining module 68 also includes validation programs to validate thetraining. For example, the training module can ascertain that sufficientdata was available and that the machine learning algorithm resulted in amodel which has a reasonable confidence value. In addition, the trainingmodule 68 can perform back testing and cross-correlation to validatemodels.

Finally, the calculation module 69 can take processed data as input,apply it in a model which has been trained by the training module 68,and calculate model results 72.

From the workstation 15, model parameters are set. Model parameters 72may be defined by reference to a pre-defined model 63 stored in thestorage device 13. For example, it is possible to refer to somebodyelse's stored model (which has been made available for use by others) orto refer to an ensemble of a number of pre-defined models. Such modelparameters are also referred to as a “basal model” to indicate that themodel parameters reference an existing, pre-defined model. In anensemble calculation, the decision function of each model is providedwith the same input and for each model a classification is calculated.Then the classifications are processed to generate an ensembleclassification. The ensemble classification can be a weighted average ofthe classifications. It can also be a median or majority value of theclassifications.

The model parameters 72 also can determine which data is to be used intraining the model. Any combination of the selections in steps 41-45 ofFIG. 4 can form model parameters 72. The model parameters 72 are used bythe data selector 65 to determine which data points to use (see alsodata point selection step 41). The model parameters can indicate one ormore specific sets of data points to be used.

For example, the model parameters may indicate that only imaging datafrom male patients of a certain age and with certain smoking habits areto be included in the training set. In this manner, the operator of theworkstation 15 can create an ad-hoc model which is completely tailoredto the circumstances of a specific patient or a specific researchquestion. In another example, the operator chooses a set of data pointsfrom a certain colleague or institution to be used in the calculation.Being able to identify specific data point sets as model parameters hasthe advantages that the operator is no longer limited to his own set ofdata points (which may contain too few data points for a good training,as shown in FIG. 5A) while still offering complete and exact controlover the model's input. The operator can make sure to only choose datapoints from trusted sources.

Likewise, the model parameters 72 can also specify which features are tobe used by the model. In general, there are many feature points that canbe calculated from imaging data. However, not all feature points areuseful in all applications of machine learning. Therefore, a goodselection of features to be used in the machine learning procedure isvery important for a successful model. Again, the operator may refer tochoices made by others, so that he or she can draw on the expertise ofothers who may have more experience.

Finally, the model parameters can specify how the training is to beperformed or which cached training data is to be used in thecalculation.

The calculation module 69 sends the model results 73 to the workstation15 for inspection by the professional. If applicable, the professionalwill formulate model feedback information 74, which is subsequentlystored in storage 13. Later, the thus stored feedback information 62 canbe used in training new models.

FIG. 6B schematically shows a method of adding new data points to thecomputation platform. In step 75 the user submits the feature vector forthe new imaging data, which was classified by the model, together withthe final classification (feedback) to the computation platform forinclusion as a data point in a data set. The new data point can be addedto an existing set of classified data points (for example a set that wasindicated by the model parameters) or a newly formed set. In step 76,the submitted data point is reviewed by a peer user. This review mayinclude a further check of the classification, to make sure that the setof classified data points is not polluted with erroneous data. If thereview is positive, the data point is added to one or more sets ofclassified data points in step 77.

FIG. 6C schematically shows a similar method for submitting modelparameters, for example for inclusion in an operator menu as shown inFIG. 7A, 8 or 9. In step 78, the model parameters are submitted. A peerreview of the parameters takes place in step 79. If the review ispositive, the model parameters are added as an operator menu item instep 80.

FIGS. 7A and 7B schematically illustrate a method for calculating anensemble model result. In step 81, models 85 are selected, for examplemodels 1, 2, . . . N. In step 82, these models 85 are provided with thesame feature vector 84, and the N respective results 86 are calculated.In step 83, an ensemble result 87 is calculated based on the N results86. There are various ways to calculate an ensemble result 87. Amajority or median result can be taken. Alternatively, a weightedaverage can be calculated as the ensemble result. Feedback 88 concerningthe ensemble result 87 can be sent to one or more of the models 85. Inthe present example, the feedback (which can be a final classificationmade by a professional based on the ensemble result) is fed back tomodel 1. As a result, the decision function of model 1 is incrementallyupdated to include feedback 88.

The computation platform storage 13 can thus store multiple models, andthe calculation module 69 of computation device 14 can apply the samefeature vector to a plurality of the models, after which the calculationmodule 69 calculates an ensemble result.

Workstation Interface

FIGS. 8A-8D show example screens for setting model parameters accordingto embodiments of the disclosed subject matter. FIG. 8A shows an examplescreen 91 of model parameters for the professional at the workstation 15to specify. These parameters include source options, modality options,image features used, patient characteristics, and data origin. Theworkstation software will usually allow to use “best practice” values asdefaults. The “best practice” values can be obtained from thecomputation platform which can maintain a database of model parametersand settings which show good performance in back-testing andcross-validation.

By specifying source options, the skilled person can indicate whichscanning devices or which modalities are to be used in the training datafor the model calculation. Modality options provides further selectionsfor the used modality. By selecting image features, the professional canselect which types of features are to be used by the model. There aregenerally many types of (visual) features that can be calculated fromimaging data, but not all features are useful for every diagnosticapplication. It is an feature of the present disclosure thatprofessionals can select which image features to use for the modelcalculation and moreover, these settings which become part of a basalmodel used can be shared among professionals so that a best of breedmodel can emerge through cooperation.

FIG. 8B shows a screen 92 for selecting which reference data sets are tobe used for training and FIG. 8C shows a screen 93 for selecting whichmodality to use in case the input has multiple modalities. Any variationon the way imaging data is gathered can be a modality. For example,using a contrasting agent before a scan is a modality. Differentequipment for imaging (even if the physical principle is the same) canbe a modality too. By allowing a granular control over modalities, andthus distinguishing between the different imaging options, it becomespossible to correlate accuracy (for any given type of diagnostic) to themodality. For example, it may be that certain types of contrasting agentexcel for certain types of detections, but hardly contribute to others.Likewise, equipment from different manufacturers may have differentstrengths and weaknesses.

FIG. 8D shows an example display 94 of a simplified screen for selectinga basal model. In display 94 the professional can select to use a basalmodel provided by Dr A, a model provided by Dr B, or a model provided byDr C. In this manner the professional can quickly select modelparameters that have been previously been used successfully. The userinterface may present on request further information about the models,for example the type of sub-application and the results of back-testingand cross validation. The further information can also comprisereferences to professional literature which describe the model andresults obtained with the model.

FIG. 9 schematically shows a display 95 for selecting an ensemblecalculation. The professional can now define a model (here called“virtual radiologist”) by combining various different models (herelabelled “Dr.A”, “Dr.B”, etc). In a typical example, each of the modelsis provided and maintained by a professional; dr. A, dr. B, etc. In thepresent example of display 95 the ensemble result of the virtualradiologist is formed by a weighted average of the models of Drs Athrough F. By combining various models in an ensemble result, theresults may be more reliable than any one of the single results. Bykeeping the various models separated from each other (until the ensembleresult is calculated), it remains possible for the professional toperform a quality control. For example, if it turns out that Dr.Bapplies a methodology that the professional does not agree with, it ispossible to remove the Dr.B's model (and underlying data) from theensemble result.

FIG. 10 schematically shows a display 100 for selecting image featuresfor use in a model. The professional can make the computation platform12 use default values by choosing the current “best practice” values.The professional can also specify exactly which imaging features to useand, if applicable, which weights these features should have in themodel.

FIG. 11 shows a flow diagram for a model calculation according toembodiments of the disclosed subject matter. In step 131. The userselects a basal model or specifies model parameters. In step 132. Theuser selects which data is to be used for optimisation. For example, theuser can select that data. According to certain patient characteristicssuch as for example age, habits, and/or gender is used for optimisation.The platform 12 (e.g. the training module 68) will then run a back testand cross validation using the selected basal model and the selecteddataset for optimisation (step 133).

FIG. 12 shows a flow diagram for a model calculation, according tofurther embodiments of the disclosed subject matter. In step 141,imaging data representing a long is segmented. The results can be shownon a display of the work station, for inspection and verification. Theimaging data can be 2-D or 3-D data. In step 142 nodules are detected togenerate a list of so-called nodule candidates. Again, these candidatescan be shown on the work station for inspection and verification.Finally, in step 143. A support vector machine (SVM) is used to decidewhether the nodule candidates are malignant or not. The professional atthe work station can inspect the classification and send feedbackinformation to the computation platform.

Modalities

FIG. 13 schematically shows a flow diagram for determining an increasein model confidence if a second modality is measured. First, in step110, the model is trained as described before using a first and secondmodality. The first modality can for example be an echo image and thesecond modality can be an MRI image. It will be recalled that a distincttype of scan (MRI, Echo, X-Ray CT, low-dose X-Ray CT, CT with contrastagent X) can be defined as a modality. In this context, also a biopsyprocedure (for determining the nature of a nodule) can be considered amodality.

In step 111 the computation platform 12 makes calculations necessary tomarginalise a second modality. Then for measurement data according thefirst modality, the model in the computation platform 12 will be able toestimate the increase in confidence if a second modality is measured. Ifthere are multiple options for a second modality, the model will be ableto estimate which modality of the multiple options will likely bring themost new information.

This advantageously makes it possible to use the model to determinewhether or not a new test (e.g. a different type of scan or even abiopsy) will bring statistically relevant new information, in step 112.Since each new test brings discomfort for the patient as well as afinancial burden, it is very useful to be able to determine whether ornot such a new test would likely improve the quality/reliability of thediagnosis in a significant way.

FIG. 14 schematically shows a method for determining whether or not anew test will bring statistically relevant new information. Theclassified feature vectors 113, used for the training of the model,include feature data from at least two two modalities (modalities 1 and2). The input data feature vector 114 misses at least one of the atleast two modalities. The features based on the second modality could beset to 0 in unclassified feature vector 114. Now the model output 116can determine not only a classification (and corresponding confidencevalue) based on the feature vector of the input data and the data pointsused for training, but it can also determine an expected increase inconfidence value if the at least one missing modality is added.

In case more than one modality is absent in the input data, the modelcan determine which modality would, in the present case, bring the moststatistically relevant additional information. In other words, whichmodality would improve the diagnosis the most.

FIG. 15A schematically shows two scenarios using respective curves 117and 118. The x-axis represents the modality 1 features (here representedas a one-dimensional feature, however this explanation holds also forthe more general case with N modality 1 features). The curves 117, 118represent, for two different models, the classification of the modelbased on the value of the modality 1 features. A classification valueclose to −1 or +1 is a high-confidence classification. A value in themiddle between −1 and +1 is a low-confidence value.

Dashed line 119 represents an exemplary value of the modality 1 featuresof new imaging data which is to be classified. In the case of the firstmodel (with curve 117) the classification will be +1 with a high degreeof confidence. There is in that case little use in measuring a secondmodality—this will be unlikely to influence the obtained classification.

However, in the case of the second model (with curve 118), theclassification will be quite uncertain. In this case, it may be usefulto measure a second modality. The computation platform can calculate alikelihood that the additional modality will provide furtherinformation.

This process can be explained graphically using FIGS. 15B and 15C. FIG.15B represents the curve for the classifications according modality 2features (which are yet unmeasured) given that the modality 1 featuresvalue is established at the value denoted by dotted line 119 if FIG.15A. FIG. 15C represents the curve according to modality 3 features(also not yet measured), with the same established value of modality 1features. In this case, it is clear that only extreme values of themodality 2 features will give a clear −1 or +1 classification. Incontrast, almost all values of the modality 3 features will give a clearclassification. Therefore, modality 3 is most likely to give usefulinformation given the currently known modality 1 features. Accordingly,the professional can propose to obtain the modality 3 information andforgo the modality 2 test.

Because the computation platform 12 has access to a large set ofprevious measurements involving various modalities, model calculationsof the computation platform can marginalise the use of any secondmodality. In this way, the model can give an estimate of the increase inconfidence if a second modality data were to become available.

It is also possible to take different outcomes (e.g. of varioustreatment options) in consideration. For example, if the data in thecentral platform is tagged with the later selected treatment andinformation related to the outcome thereof, it is possible to use thepast experiences to improve choices in the future.

FIG. 15D illustrates schematically a number of outcomes for patientswho, after a diagnosis involving a measurement of modality 1 features,received treatment A. Again, the modality 1 features are represented,for clarity only, as one-dimensional data along the horizontal axis. Thevertical axis is in this case the outcome variable, which is the numberof months of life after treatment A. Each outcome is represented as asquare 122 at the intersection of the modality 1 feature value and theoutcome. The same is shown in FIG. 15E, for treatment B with squares123.

A clear pattern can be distinguished when comparing FIGS. 15D and 15E:while treatment B is about equally effective across all previouslymeasured values of modality 1, treatment A shows a statisticaldependency, where higher modality 1 values lead to lower outcome values.For both treatments, the average outcome is more or less equal (about 18months). However, for a patient with modality 1 features according toline 119, treatment A has statistically a better outcome than treatmentB.

Based on tagged patient data, the system is thus able, when there is astatistically significant relation between a modality and a treatment'soutcome, indicate an advantageous course of treatment, even if theoverall statistics of the treatments do not give a clear direction.

FIG. 15F shows a further scenario, wherein two modalities areconsidered. The outcome (again the number of months of life expectancyafter a treatment) is now shown as a label for each point 124. Nowassume that only modality 1 is measured. In the present case, there isno clear statistical relation between modality 1 features and theoutcome, so that based on previous experiences, the outcome of thetreatment could be anything between 12 and 48 months. However, when thepotential modality 2 features are also considered, a statistical linkemerges, where higher values of modality 2 features correlate withhigher outcomes. In the present example, it might thus make sense tomeasure modality 2 in order to get a better estimate of a futuretreatment's outcome and thus make a better decision about treatment.

Patient Data

FIG. 16 schematically shows a flow diagram for making patient dataavailable to the computation platform 12. In step 125 a patient selectsdata privacy options for example through options on a paper form, or inan interactive computer application. In step 126 data is measured andstored in local storage 11, as described above. Patient metadata can bestored in a further local storage 11 a.

In step 127, it is determined whether data may be uploaded depending onthe selected privacy options. For example, the options may specify thatonly certain types of data are allowed for uploads. The options may alsospecify that the data must be made anonymous. The options may specifythat certain aspects of the data are to be removed before uploading.Depending on the outcome of the check in step 127, in step 128 data isuploaded to the computation platform 12. By integrating the handling ofpatient privacy in the upload flow for the computation platform 12, itis possible to efficiently upload data from a wide variety of sources tothe computation platform 12, while maintaining the option to excludecertain types of data if the patient's preferences or the regulations sorequire.

CONCLUDING REMARKS

In the foregoing description of the figures, aspects of the disclosurehave been described with reference to specific embodiments. It will,however, be evident that various modifications and changes may be madethereto without departing from the scope of the disclosure as summarizedin the attached claims.

In addition, many modifications may be made to adapt a particularsituation or material to the teachings of the disclosure withoutdeparting from the essential scope thereof. Therefore, it is intendedthat the disclosure not be limited to the particular embodimentsdisclosed, but that the disclosure will include all embodiments fallingwithin the scope of the appended claims.

It is also noted that when items are presented in a single drawn box inthe figure, this is but a logical representation. In many real-worldimplementations, a plurality of such “boxes” can be implemented in asingle chip or server, or functionality which is represented in a singlebox may be distributed (e.g. parallelized) over a number of differentplatforms. The skilled person is aware of such minor variations that areopen to the implementer of this disclosure. The attached claims are notto be read as limited to specific segmentation of functionalities shownin the figures.

Combinations of specific features of various aspects of the disclosuremay be made. An aspect of the disclosure may be further advantageouslyenhanced by adding a feature that was described in relation to anotheraspect of the disclosure.

It is to be understood that the disclosure is limited by the annexedclaims and its technical equivalents only. In this document and in itsclaims, the verb “to comprise” and its conjugations are used in theirnon-limiting sense to mean that items following the word are included,without excluding items not specifically mentioned. In addition,reference to an element by the indefinite article “a” or “an” does notexclude the possibility that more than one of the element is present,unless the context clearly requires that there be one and only one ofthe elements. The indefinite article “a” or “an” thus usually means “atleast one”.

The invention claimed is:
 1. Method for classifying a region of interestin imaging data, the method comprising: providing imaging data measuredusing a first modality, said imaging data excluding data measured usinga second modality; calculating a feature vector for at least one regionof interest in the imaging data, said feature vector including featurescalculated from the provided imaging data; projecting the feature vectorfor the at least one region of interest in the imaging data using adecision function to generate a classification, wherein the decisionfunction is based on classified feature vectors including features ofthe first modality and features of at least the second modality;calculating a confidence value of the classification; and estimating anincrease in the confidence value of the classification if the featurevector were enhanced with features of the second modality calculatedfrom imaging data measured using the second modality.
 2. The methodaccording to claim 1, further comprising determining an outcome valuebased on outcome values associated with the classified feature vectors.3. The method according to claim 1, further comprising marginalizing thedecision function for features of the second modality.
 4. The methodaccording to claim 3, further comprising marginalizing the decisionfunction for features of a third modality.
 5. The method according toclaim 4, further comprising selecting one of the second modality and thethird modality.
 6. The method according to claim 4, wherein the decisionfunction has been trained using the respective set of classified featurevectors to project a feature vector to a classification.
 7. The methodaccording to claim 6, wherein the decision function is based on one of asupport vector machine (SVM), a decision tree, or a boosted stump. 8.The method according to claim 7, wherein the imaging data represents ahuman lung or a human breast.
 9. The method according to claim 8,wherein the imaging data is a computer tomography (CT) image.
 10. Themethod according to claim 9, wherein the first, second, or thirdmodality is characterized by a type of contrasting agent used inobtaining the imaging data.
 11. The method according to claim 9, whereinthe first, second, or third modality is characterized by a type ofequipment used in obtaining the imaging data.
 12. The method accordingto claim 1, further comprising determining and presenting arecommendation for obtaining second modality features.
 13. The methodaccording to claim 12, wherein the decision function is based on one ofa support vector machine (SVM), a decision tree, or a boosted stump. 14.Computation platform comprising a computation device, wherein thecomputation device is adapted for: receiving imaging data measured usinga first modality, said imaging data excluding data measured using asecond modality; calculating a feature vector for at least one region ofinterest in the imaging data, said feature vector including featurescalculated from the received imaging data; projecting the feature vectorfor the at least one region of interest in the imaging data using adecision function to generate a classification, wherein the decisionfunction is based on classified feature vectors including features ofthe first modality and features of at least the second modality;calculating a confidence value of the classification; and estimating anincrease in the confidence value of the classification if the featurevector were enhanced with features of the second modality calculatedfrom imaging data measured using the second modality.
 15. The platformaccording to claim 14, further adapted for determining an outcome valuebased on outcome values associated with the classified feature vectors.16. Non-transitory computer readable medium comprising computerinstructions for implementing the method according to claim 1.